With the increasing ability of large language models (LLMs), in-context learning (ICL) has become a new paradigm for natural language processing (NLP), where LLMs make predictions only based on contexts augmented with a few training examples. It has been a new trend exploring ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress, challenges, and future work in ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques of ICL, including training strategies, prompting strategies, and so on. Finally, we present the challenges of ICL and provide potential directions for further research. We hope our work can encourage more research on uncovering how ICL works and improving ICL in future work.

Zero-Shot Learning has been a highlighted research topic in both vision and language areas. Recently, most existing methods adopt structured knowledge information to model explicit correlations among categories and use deep graph convolutional network to propagate information between different categories. However, it is difficult to add new categories to existing structured knowledge graph, and deep graph convolutional network suffers from over-smoothing problem. In this paper, we provide a new semantic enhanced knowledge graph that contains both expert knowledge and categories semantic correlation. Our semantic enhanced knowledge graph can further enhance the correlations among categories and make it easy to absorb new categories. To propagate information on the knowledge graph, we propose a novel Residual Graph Convolutional Network (ResGCN), which can effectively alleviate the problem of over-smoothing. Experiments conducted on the widely used large-scale ImageNet-21K dataset and AWA2 dataset show the effectiveness of our method, and establish a new state-of-the-art on zero-shot learning. Moreover, our results on the large-scale ImageNet-21K with various feature extraction networks show that our method has better generalization and robustness.

Traditional multilingual neural machine translation (MNMT) uses a single model to translate all directions. However, with the increasing scale of language pairs, simply using a single model for massive MNMT brings new challenges: parameter tension and large computations. In this paper, we revisit multi-way structures by assigning an individual branch for each language (group). Despite being a simple architecture, it is challenging to train de-centralized models due to the lack of constraints to align representations from all languages. We propose a localized training recipe to map different branches into a unified space, resulting in an efficient detachable model, Lego-MT. For a fair comparison, we collect data from OPUS and build the first large-scale open-source translation benchmark covering 7 language-centric data, each containing 445 language pairs. Experiments show that Lego-MT (1.2B) brings gains of more than 4 BLEU while outperforming M2M-100 (12B) (We will public all training data, models, and checkpoints)

With increasing scale, large language models demonstrate both quantitative improvement and new qualitative capabilities, especially as zero-shot learners, like GPT-3. However, these results rely heavily on delicate prompt design and large computation. In this work, we explore whether the strong zero-shot ability could be achieved at a smaller model scale without any external supervised data. To achieve this goal, we revisit masked language modeling and present a geometry-guided self-supervised learning method (Go-tuningfor short) by taking a small number of task-aware self-supervised data to update language models further. Experiments show that Go-tuning can enable T5-small (80M) competitive zero-shot results compared with large language models, such as T5-XL (3B). We also apply Go-tuning on multi-task settings and develop a multi-task model, mgo-T5 (250M). It can reach the average performance of OPT (175B) on 9 datasets.

Answering complex questions over textual resources remains a challenging problem$\unicode{x2013}$especially when interpreting the fine-grained relationships among multiple entities that occur within a natural-language question or clue. Curated knowledge bases (KBs), such as YAGO, DBpedia, Freebase and Wikidata, have been widely used in this context and gained great acceptance for question-answering (QA) applications in the past decade. While current KBs offer a concise representation of structured knowledge, they lack the variety of formulations and semantic nuances as well as the context of information provided by the natural-language sources. With BigText-QA, we aim to develop an integrated QA system which is able to answer questions based on a more redundant form of a knowledge graph (KG) that organizes both structured and unstructured (i.e., "hybrid") knowledge in a unified graphical representation. BigText-QA thereby is able to combine the best of both worlds$\unicode{x2013}$a canonical set of named entities, mapped to a structured background KB (such as YAGO or Wikidata), as well as an open set of textual clauses providing highly diversified relational paraphrases with rich context information.

ASR can be improved by multi-task learning (MTL) with domain enhancing or domain adversarial training, which are two opposite objectives with the aim to increase/decrease domain variance towards domain-aware/agnostic ASR, respectively. In this work, we study how to best apply these two opposite objectives with speaker labels to improve conformer-based ASR. We also propose a novel adaptive gradient reversal layer for stable and effective adversarial training without tuning effort. Detailed analysis and experimental verification are conducted to show the optimal positions in the ASR neural network (NN) to apply speaker enhancing and adversarial training. We also explore their combination for further improvement, achieving the same performance as i-vectors plus adversarial training. Our best speaker-based MTL achieves 7\% relative improvement on the Switchboard Hub5'00 set. We also investigate the effect of such speaker-based MTL w.r.t. cleaner dataset and weaker ASR NN.

在本文中，我们研究了神经视频压缩（NVC）中位分配的问题。首先，我们揭示了最近声称是最佳的位分配方法实际上是由于其实施而是最佳的。具体而言，我们发现其亚典型性在于半损坏的变异推理（SAVI）对潜在的不正确的应用，具有非物质变异后验。然后，我们表明，在非因素潜伏期上校正的SAVI校正版本需要递归地通过梯度上升应用后传播，这是我们得出校正后的最佳位分配算法的。由于校正位分配的计算不可行性，我们设计了有效的近似值以使其实用。经验结果表明，我们提出的校正显着改善了R-D性能和比特率误差的错误分配，并且比所有其他位分配方法都大大提高了。源代码在补充材料中提供。

演讲者的适应性对于建立强大的自动语音识别（ASR）系统很重要。在这项工作中，我们根据基于配置符号的声学模型（AM）在300H数据集中的功能空间方法研究了扬声器自适应训练（SAT）的各种方法。我们提出了一种称为加权简单添加的方法，该方法将加权的说话者信息向量添加到构象异构体AM的多头自发动模块的输入中。使用此方法用于SAT，我们在HUB5'00和HUB5'01的Callhome部分方面取得了3.5％和4.5％的相对改善。此外，我们以先前的作品为基础，在此基础上，我们为基于构象异构体的混合动力AM提出了一种新颖的竞争培训配方。我们扩展并改善了此食谱，在该配方中，我们在打电筒300H HUB5'00数据集上的单词误差（WER）方面取得了11％的相对改善。我们还通过将参数总数减少34％，从而使该配方有效。

多模式的预训练和知识发现是多模式机器学习中的两个重要研究主题。然而，没有现有的作品试图将知识发现与知识指导的多模式预训练联系起来。在本文中，我们建议将它们统一成一个连续的学习框架以进行相互改进。以图像和文本的开放域单模式数据集为输入，我们将知识图作为支持这两个任务的基础。对于知识发现，使用预训练的模型来识别图表上的跨模式链接。对于模型预训练，将知识图用作指导模型更新的外部知识。这两个步骤是在我们的持续学习框架中迭代执行的。关于知识发现和预训练模型，MS-Coco和FlickR30K的实验结果验证了我们框架的有效性。

许多现有的神经结构搜索（NAS）解决方案依赖于架构评估的下游培训，这需要巨大的计算。考虑到这些计算带来了大量碳足迹，本文旨在探索绿色（即环保）NAS解决方案，可以在不培训的情况下评估架构。直观地，由架构本身引起的梯度，直接决定收敛和泛化结果。它激励我们提出梯度内核假设：梯度可以用作下游训练的粗粒粒度，以评估随机初始化网络。为了支持假设，我们进行理论分析，找到一个实用的梯度内核，与培训损失和验证性能有良好的相关性。根据这一假设，我们提出了一种新的基于内核的架构搜索方法knas。实验表明，KNA可实现比图像分类任务的“火车-TER-TEST”范式更快地实现竞争力。此外，极低的搜索成本使其具有广泛的应用。搜索网络还优于两个文本分类任务的强大基线Roberta-Light。代码可用于\ url {https://github.com/jingjing-nlp/knas}。